Apply LERC valid-mask in GPU decode path (depends on #1529)#1535
Open
brendancol wants to merge 2 commits intoxarray-contrib:mainfrom
Open
Apply LERC valid-mask in GPU decode path (depends on #1529)#1535brendancol wants to merge 2 commits intoxarray-contrib:mainfrom
brendancol wants to merge 2 commits intoxarray-contrib:mainfrom
Conversation
lerc.decode returns (rc, data, valid_mask, ...) but the wrapper only forwarded data.tobytes(). GDAL writes LERC TIFFs whose masked pixels are zero-filled in the data array, so downstream code that masks by nodata silently sees those zeros as real measurements. What changed: _compression.py: new lerc_decompress_with_mask(data) -> (bytes, valid_mask_or_None). lerc_decompress now calls it and drops the mask, preserving its existing signature. An all-True mask collapses to None so callers skip the fill pass. _reader.py: _decode_strip_or_tile takes a separate path for LERC, calls the new wrapper, and writes nodata into masked positions after reshape. A small _resolve_masked_fill helper reads ifd.nodata_str and falls back to NaN for float dtypes or 0 for integer dtypes when no GDAL_NODATA is set. Each call site (strip reader, tile reader, COG-over-HTTP reader) passes a precomputed masked_fill. tests/test_lerc_valid_mask.py: 8 new tests. Wrapper tests cover the no-mask path, the all-True-mask collapse, and a partial mask. TIFF round-trip tests cover float32 with NaN nodata, float32 with -9999, uint16 with 65535, and a regression that an unmasked LERC file still round-trips bit-exact. The round-trip tests monkeypatch lerc_compress to inject a per-tile mask through a predicate, since the writer hard-codes hasMask=False. Option A was chosen (plumb nodata into the decode path). Option B would have required changing decompress()'s return shape, which has callers in tests and in _gpu_decode.py. Out of scope: the GPU LERC path in _gpu_decode.py still drops the mask. The constraints excluded changes there.
) The CPU LERC reader from xarray-contrib#1529 honours the LERC valid-mask and writes the file's nodata sentinel into masked pixels. The GPU LERC tile-decode path was still dropping the mask, so masked pixels read back as 0 on GPU but as NaN or the sentinel on CPU. Same bug, GPU side. Changes: _gpu_decode.py: the LERC branch now calls lerc_decompress_with_mask per tile and keeps any returned valid-mask. After predictor decode and tile assembly, _apply_lerc_mask_fill builds an invalid mask on host (matching the GPU assembly kernel's tile-grid layout), copies it to GPU once, and overwrites masked positions with the resolved fill value. Tiles LERC reports as fully valid skip the host work, so the no-mask path stays zero-copy. gpu_decode_tiles and gpu_decode_tiles_from_file get a masked_fill keyword that is forwarded through. read_geotiff_gpu computes it via _resolve_masked_fill(ifd.nodata_str, file_dtype) for LERC sources. tests/test_lerc_valid_mask_gpu.py: 4 tests covering float32+NaN, float32+sentinel, uint16+sentinel, and the no-mask regression. Each compares read_geotiff_gpu output to read_to_array output for the same file. Skipped unless cupy + CUDA + lerc are available. Out of scope: the encode side. The xrspatial writer still hard-codes hasMask=False; the tests reuse the lerc_compress monkeypatch fixture from the CPU PR to inject a valid-mask through lerc.encode directly.
Contributor
Author
|
@copilot review |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Follow-up to #1529. The CPU LERC reader applies the LERC valid-mask
and writes nodata into masked positions, but the GPU LERC tile-decode
path in
_gpu_decode.pywas still dropping the mask. A masked pixelread back as 0 on GPU and as NaN or the sentinel on CPU.
This PR fixes the GPU side:
lerc_decompress_with_maskper tile andkeeps any returned mask.
built on host, copied to the GPU once, and used to write the
resolved fill value into masked positions.
gpu_decode_tilesandgpu_decode_tiles_from_fileget amasked_fill=kwarg thatread_geotiff_gpupopulates via_resolve_masked_fill(ifd.nodata_str, file_dtype)when compressionis LERC.
Depends on #1529 (uses
lerc_decompress_with_maskand_resolve_masked_fillintroduced there). Land #1529 first; this PRshould rebase cleanly off main once it is merged.
Test plan
pytest xrspatial/geotiff/tests/test_lerc_valid_mask_gpu.pypasses (4 tests).pytest xrspatial/geotiff/tests/test_lerc.py test_lerc_max_z_error.py test_gpu_byteswap_1508.py test_lerc_valid_mask.pypasses (38 tests).read_geotiff_gpureturns NaN at the masked position (matches CPU); without the fix it returned 0.0.